System and method for determining input object information from proximity and force measurements

ABSTRACT

A processing system for an input device having a sensing region overlapping an input surface and an array of sensor electrodes configured to form a plurality of proximity pixels and a plurality of force pixels. The processing system is configured to: determine a proximity image indicative of positional information for input objects; determine a force image indicative of local deflection of the input surface in response to force applied by the input objects; determine a respective group of proximity pixels from the proximity image corresponding to each input object; determine a respective group of force pixels from the force image corresponding to each determined group of proximity pixels; determine the position of each input object based on the determined groups of proximity pixels; and determine the force associated with each input object based on the determined groups of force pixels.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation application of and, thereby,claims benefit under 35 U.S.C. §120 to U.S. application Ser. No.14/320,005, entitled, “SYSTEM AND METHOD FOR DETERMINING INPUT OBJECTINFORMATION FROM PROXIMITY AND FORCE MEASUREMENTS,” filed on Jun. 30,2014, and incorporated herein by reference.

FIELD OF THE INVENTION

This invention generally relates to electronic devices, and morespecifically relates to sensor devices and using sensor devices forproducing user interface inputs.

BACKGROUND OF THE INVENTION

Proximity sensor devices (also commonly called touch sensor devices) arewidely used in a variety of electronic systems. A proximity sensordevice typically includes a sensing region, often demarked by a surface,in which input objects may be detected. Example input objects includefingers, styli, and the like. The proximity sensor device may utilizeone or more sensors based on capacitive, resistive, inductive, optical,acoustic and/or other technology. Further, the proximity sensor devicemay determine the presence, location and/or motion of a single inputobject in the sensing region, or of multiple input objectssimultaneously in the sensor region.

The proximity sensor device may be used to enable control of anassociated electronic system. For example, proximity sensor devices areoften used as input devices for larger computing systems, including:notebook computers and desktop computers. Proximity sensor devices arealso often used in smaller systems, including: handheld systems such aspersonal digital assistants (PDAs), remote controls, and communicationsystems such as wireless telephones and text messaging systems.Increasingly, proximity sensor devices are used in media systems, suchas CD, DVD, MP3, video or other media recorders or players. Theproximity sensor device may be integral or peripheral to the computingsystem with which it interacts.

Presently known sensor devices have the ability to detect both theposition and force associated with objects in the sensing region.However, reliably determining the amount of force applied by multipleobjects, respectively, is a continuing challenge. This limits theflexibility of the proximity sensor device in providing different typesof user interface actions in response to different numbers of objects orgestures with different numbers of objects.

Thus, improved techniques are needed for reliably determining therespective force applied by multiple objects in a sensing region of aproximity sensor device. Other desirable features and characteristicswill become apparent from the subsequent detailed description and theappended claims, taken in conjunction with the accompanying drawings andthe foregoing technical field and background.

BRIEF SUMMARY OF THE INVENTION

Devices and methods are provided for facilitating improved sensor deviceusability by determining the respective force applied by each of aplurality of input objects in a sensing region of a capacitive sensor.In particular, various embodiments determine a proximity (touch) imageand a force image associated with one or more input objects. The pixelswhich make up the touch image may then be segmented into individualtouch “blobs”, each comprising a list of touch pixels corresponding to aunique input object. Touch image segmentation typically involves apixel-by-pixel approach using, for example, a watershed or analogousalgorithm. If the touch image is segmented correctly, each touch blobwill reliably correspond to a unique input object (e.g., finger).

The force image is then segmented into individual force blobs, eachcomprising a subset of the pixels which make up the force image. If theforce image is segmented correctly, each force blob will also reliablycorrespond to a unique input object. To help ensure that force imagesegmentation proceeds in a manner consistent with the previouslyperformed touch image segmentation, the touch blobs are used insegmenting the force image. That is, the pixels associated with eachtouch blob are used to define an initial force blob. The initial forceblobs are then processed using any suitable image segmentationtechnique. In an embodiment, the force pixels may be aligned exactlywith the touch pixels in a one-to-one correspondence, simplifying theforce image segmentation process. Alternatively, the force pixels andtouch pixels may be configured in a greater than or less than one-to-onecorrespondence.

The force blobs may then be used to reliably determine the amount offorce associated with each input object, thereby improving sensor deviceusability.

BRIEF DESCRIPTION OF THE DRAWINGS

The preferred exemplary embodiment of the present invention willhereinafter be described in conjunction with the appended drawings,where like designations denote like elements, and wherein:

FIG. 1 is a block diagram of an exemplary system that includes an inputdevice in accordance with an embodiment of the invention;

FIG. 2 is a schematic view of an exemplary electrode array illustratingtouch sensor and/or force sensor pixels in accordance with an embodimentof the invention;

FIG. 3 is a schematic side elevation view of a conceptual layout for acombination touch and force sensor stack-up including touch sensorelectrodes and force sensor electrodes in accordance with an embodimentof the invention;

FIG. 4 is a top view of an input device with multiple input objects inthe sensing region in accordance with an embodiment of the invention;

FIG. 5 is a side view of the input device of FIG. 4 showing thedirection of applied force by one or more input objects upon an inputsurface of the sensing region in accordance with an embodiment of theinvention;

FIG. 6 is a schematic top view an input device graphically illustratingtouch image information and associated segmented touch blobs formultiple objects in the sensing region in accordance with an embodimentof the invention;

FIG. 7 is a schematic top view of an input device illustrating forceimage information for the input objects shown of FIG. 6 in accordancewith an embodiment of the invention;

FIG. 8 is a two-dimensional graphic representation of exemplary touchbasins illustrating a touch segmentation algorithm in accordance with anembodiment of the invention;

FIG. 9 is a schematic top view of an input device illustrating forceimage information superimposed on touch blobs illustrating initialseeding of touch blob information into initial force basins for theinput objects shown of FIGS. 6 and 7 in accordance with an embodiment ofthe invention;

FIG. 10 is a two-dimensional graphic representation of touch blobsseeding force basins in the context of an exemplary force imagesegmentation algorithm in accordance with an embodiment of theinvention;

FIG. 11 is a two-dimensional graphic illustration of a first force blobof greater extent than its corresponding touch blob, and a second forceblob of lesser extent than its corresponding touch blob in accordancewith an embodiment of the invention; and

FIG. 12 is a flow diagram of an exemplary force image segmentationprocess in accordance with an embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

The following detailed description is merely exemplary in nature and isnot intended to limit the invention or the application and uses of theinvention. Furthermore, there is no intention to be bound by anyexpressed or implied theory presented in the preceding technical field,background, brief summary or the following detailed description.

The embodiments of the present invention provide a device and methodthat facilitates improved sensor device usability. Specifically, thedevice and method provide improved device usability by facilitating thereliable determination of the amount of force per input object for oneor more objects in a sensing region of a capacitive sensor device. Invarious embodiments, the results of a touch image segmentation algorithmare used to seed a subsequent force image segmentation algorithm, tothereby reliably correlate the segmented force image to the respectiveindividual input objects determined during touch image segmentation.

Turning now to the figures, FIG. 1 is a block diagram of an exemplaryinput device 100, in accordance with embodiments of the invention. Theinput device 100 may be configured to provide input to an electronicsystem (not shown). As used in this document, the term “electronicsystem” (or “electronic device”) broadly refers to any system capable ofelectronically processing information. Some non-limiting examples ofelectronic systems include personal computers of all sizes and shapes,such as desktop computers, laptop computers, netbook computers, tablets,web browsers, e-book readers, and personal digital assistants (PDAs).Additional example electronic systems include composite input devices,such as physical keyboards that include input device 100 and separatejoysticks or key switches. Further example electronic systems includeperipherals such as data input devices (including remote controls andmice), and data output devices (including display screens and printers).Other examples include remote terminals, kiosks, and video game machines(e.g., video game consoles, portable gaming devices, and the like).Other examples include communication devices (including cellular phones,such as smart phones), and media devices (including recorders, editors,and players such as televisions, set-top boxes, music players, digitalphoto frames, and digital cameras). Additionally, the electronic systemcould be a host or a slave to the input device.

The input device 100 can be implemented as a physical part of theelectronic system, or can be physically separate from the electronicsystem. As appropriate, the input device 100 may communicate with partsof the electronic system using any one or more of the following: buses,networks, and other wired or wireless interconnections. Examples includeI2C, SPI, PS/2, Universal Serial Bus (USB), Bluetooth, RF, and IRDA.

In FIG. 1, the input device 100 is shown as a proximity sensor device(also often referred to as a “touchpad” or a “touch sensor device”)configured to sense input provided by one or more input objects 140 in asensing region 120. Example input objects include fingers and styli, asshown in FIG. 1.

Sensing region 120 encompasses any space above, around, in and/or nearthe input device 100 in which the input device 100 is able to detectuser input (e.g., user input provided by one or more input objects 140).The sizes, shapes, and locations of particular sensing regions may varywidely from embodiment to embodiment. In some embodiments, the sensingregion 120 extends from a surface of the input device 100 in one or moredirections into space until signal-to-noise ratios prevent sufficientlyaccurate object detection. The distance to which this sensing region 120extends in a particular direction, in various embodiments, may be on theorder of less than a millimeter, millimeters, centimeters, or more, andmay vary significantly with the type of sensing technology used and theaccuracy desired. Thus, some embodiments sense input that comprises nocontact with any surfaces of the input device 100, contact with an inputsurface (e.g. a touch surface) of the input device 100, contact with aninput surface of the input device 100 coupled with some amount ofapplied force or pressure, and/or a combination thereof. In variousembodiments, input surfaces may be provided by surfaces of casingswithin which the sensor electrodes reside, by face sheets applied overthe sensor electrodes or any casings, etc. In some embodiments, thesensing region 120 has a rectangular shape when projected onto an inputsurface of the input device 100.

The input device 100 may utilize any combination of sensor componentsand sensing technologies to detect user input in the sensing region 120.The input device 100 comprises one or more sensing elements fordetecting user input. As several non-limiting examples, the input device100 may use capacitive, elastive, resistive, inductive, magnetic,acoustic, ultrasonic, and/or optical techniques.

Some implementations are configured to provide images that span one,two, three, or higher dimensional spaces. Some implementations areconfigured to provide projections of input along particular axes orplanes.

In some resistive implementations of the input device 100, a flexibleand conductive first layer is separated by one or more spacer elementsfrom a conductive second layer. During operation, one or more voltagegradients are created across the layers. Pressing the flexible firstlayer may deflect it sufficiently to create electrical contact betweenthe layers, resulting in voltage outputs reflective of the point(s) ofcontact between the layers. These voltage outputs may be used todetermine positional information.

In some inductive implementations of the input device 100, one or moresensing elements pick up loop currents induced by a resonating coil orpair of coils. Some combination of the magnitude, phase, and frequencyof the currents may then be used to determine positional information.

In some capacitive implementations of the input device 100, voltage orcurrent is applied to create an electric field. Nearby input objectscause changes in the electric field, and produce detectable changes incapacitive coupling that may be detected as changes in voltage, current,or the like.

Some capacitive implementations utilize arrays or other regular orirregular patterns of capacitive sensing elements to create electricfields. In some capacitive implementations, separate sensing elementsmay be ohmically shorted together to form larger sensor electrodes. Somecapacitive implementations utilize resistive sheets, which may beuniformly resistive.

Some capacitive implementations utilize “self capacitance” (or “absolutecapacitance”) sensing methods based on changes in the capacitivecoupling between sensor electrodes and an input object. In variousembodiments, an input object near the sensor electrodes alters theelectric field near the sensor electrodes, thus changing the measuredcapacitive coupling. In one implementation, an absolute capacitancesensing method operates by modulating sensor electrodes with respect toa reference voltage (e.g. system ground), and by detecting thecapacitive coupling between the sensor electrodes and input objects.

Some capacitive implementations utilize “mutual capacitance” (or“transcapacitance”) sensing methods based on changes in the capacitivecoupling between sensor electrodes. In various embodiments, an inputobject near the sensor electrodes alters the electric field between thesensor electrodes, thus changing the measured capacitive coupling. Inone implementation, a transcapacitive sensing method operates bydetecting the capacitive coupling between one or more transmitter sensorelectrodes (also “transmitter electrodes” or “transmitters”) and one ormore receiver sensor electrodes (also “receiver electrodes” or“receivers”). Transmitter sensor electrodes may be modulated relative toa reference voltage (e.g., system ground) to transmit transmittersignals. Receiver sensor electrodes may be held substantially constantrelative to the reference voltage to facilitate receipt of resultingsignals. A resulting signal may comprise effect(s) corresponding to oneor more transmitter signals, and/or to one or more sources ofenvironmental interference (e.g. other electromagnetic signals). Sensorelectrodes may be dedicated transmitters or receivers, or may beconfigured to both transmit and receive.

In FIG. 1, a processing system 110 is shown as part of the input device100. The processing system 110 is configured to operate the hardware ofthe input device 100 to detect input in the sensing region 120. Theprocessing system 110 comprises parts of or all of one or moreintegrated circuits (ICs) and/or other circuitry components. Forexample, a processing system for a mutual capacitance sensor device maycomprise transmitter circuitry configured to transmit signals withtransmitter sensor electrodes, and/or receiver circuitry configured toreceive signals with receiver sensor electrodes). In some embodiments,the processing system 110 also comprises electronically-readableinstructions, such as firmware code, software code, and/or the like. Insome embodiments, components composing the processing system 110 arelocated together, such as near sensing element(s) of the input device100. In other embodiments, components of processing system 110 arephysically separate with one or more components close to sensingelement(s) of input device 100, and one or more components elsewhere.For example, the input device 100 may be a peripheral coupled to adesktop computer, and the processing system 110 may comprise softwareconfigured to run on a central processing unit of the desktop computerand one or more ICs (perhaps with associated firmware) separate from thecentral processing unit. As another example, the input device 100 may bephysically integrated in a phone, and the processing system 110 maycomprise circuits and firmware that are part of a main processing systemof the phone. In some embodiments, the processing system 110 isdedicated to implementing the input device 100. In other embodiments,the processing system 110 also performs other functions, such asoperating display screens, driving haptic actuators, etc.

The processing system 110 may be implemented as a set of modules thathandle different functions of the processing system 110. Each module maycomprise circuitry that is a part of the processing system 110,firmware, software, or a combination thereof. In various embodiments,different combinations of modules may be used. Example modules includehardware operation modules for operating hardware such as sensorelectrodes and display screens, data processing modules for processingdata such as sensor signals and positional information, and reportingmodules for reporting information. Further example modules includesensor operation modules configured to operate sensing element(s) todetect input, identification modules configured to identify gesturessuch as mode changing gestures, and mode changing modules for changingoperation modes.

In some embodiments, the processing system 110 responds to user input(or lack of user input) in the sensing region 120 directly by causingone or more actions. Example actions include changing operation modes,as well as GUI actions such as cursor movement, selection, menunavigation, and other functions. In some embodiments, the processingsystem 110 provides information about the input (or lack of input) tosome part of the electronic system (e.g. to a central processing systemof the electronic system that is separate from the processing system110, if such a separate central processing system exists). In someembodiments, some part of the electronic system processes informationreceived from the processing system 110 to act on user input, such as tofacilitate a full range of actions, including mode changing actions andGUI actions.

For example, in some embodiments, the processing system 110 operates thesensing element(s) of the input device 100 to produce electrical signalsindicative of input (or lack of input) in the sensing region 120. Theprocessing system 110 may perform any appropriate amount of processingon the electrical signals in producing the information provided to theelectronic system. For example, the processing system 110 may digitizeanalog electrical signals obtained from the sensor electrodes. Asanother example, the processing system 110 may perform filtering orother signal conditioning. As yet another example, the processing system110 may subtract or otherwise account for a baseline, such that theinformation reflects a difference between the electrical signals and thebaseline. As yet further examples, the processing system 110 maydetermine positional information, recognize inputs as commands,recognize handwriting, and the like.

“Positional information” as used herein broadly encompasses absoluteposition, relative position, velocity, acceleration, and other types ofspatial information. Exemplary “zero-dimensional” positional informationincludes near/far or contact/no contact information. Exemplary“one-dimensional” positional information includes positions along anaxis. Exemplary “two-dimensional” positional information includesmotions in a plane. Exemplary “three-dimensional” positional informationincludes instantaneous or average velocities in space. Further examplesinclude other representations of spatial information. Historical dataregarding one or more types of positional information may also bedetermined and/or stored, including, for example, historical data thattracks position, motion, or instantaneous velocity over time.

In some embodiments, the input device 100 is implemented with additionalinput components that are operated by the processing system 110 or bysome other processing system. These additional input components mayprovide redundant functionality for input in the sensing region 120, orsome other functionality. FIG. 1 shows buttons 130 near the sensingregion 120 that can be used to facilitate selection of items using theinput device 100. Other types of additional input components includesliders, balls, wheels, switches, and the like. Conversely, in someembodiments, the input device 100 may be implemented with no other inputcomponents.

In some embodiments, the input device 100 comprises a touch screeninterface, and the sensing region 120 overlaps at least part of anactive area of a display screen. For example, the input device 100 maycomprise substantially transparent sensor electrodes overlaying thedisplay screen and provide a touch screen interface for the associatedelectronic system. The display screen may be any type of dynamic displaycapable of displaying a visual interface to a user, and may include anytype of light emitting diode (LED), organic LED (OLED), cathode ray tube(CRT), liquid crystal display (LCD), plasma, electroluminescence (EL),or other display technology. The input device 100 and the display screenmay share physical elements. For example, some embodiments may utilizesome of the same electrical components for displaying and sensing. Asanother example, the display screen may be operated in part or in totalby the processing system 110.

It should be understood that while many embodiments of the invention aredescribed in the context of a fully functioning apparatus, themechanisms of the present invention are capable of being distributed asa program product (e.g., software) in a variety of forms. For example,the mechanisms of the present invention may be implemented anddistributed as a software program on information bearing media that arereadable by electronic processing systems (e.g., non-transitorycomputer-readable and/or recordable/writable information bearing mediareadable by the processing system 110). Additionally, the embodiments ofthe present invention apply equally regardless of the particular type ofmedium used to carry out the distribution. Examples of non-transitory,electronically readable media include various discs, memory sticks,memory cards, memory modules, and the like. Electronically readablemedia may be based on flash, optical, magnetic, holographic, or anyother storage technology.

FIG. 2 shows a portion of a sensor electrode pattern configured to sensethe position (or force) associated with objects proximate the pattern,according to some embodiments. For clarity of illustration anddescription, FIG. 2 shows a pattern (e.g., an array) 200 comprising aplurality of transmitter electrodes 220A-C and a plurality of receiverelectrodes 210A-D defining an array of pixels 214. In the context ofproximity sensing, the receiver electrodes 210 and transmitterelectrodes 220 function as touch sensor electrodes, which measure achange in capacitance of the array of pixels 214, producing a touchimage indicative of input objects present in the sensing region. In thecontext of force sensing, the receiver electrodes 210 and transmitterelectrodes 220 function as force sensor electrodes, which measure achange in capacitance of the array of pixels 214, producing a forceimage indicative of input objects applying a force to the sensorelectrodes.

Transmitter electrodes 220 and receiver electrodes 210 are typicallyohmically isolated from each other. That is, one or more insulatorsseparate the transmitter electrodes from the receiver electrodes andprevent them from electrically shorting to each other. In someembodiments, receiver electrodes 210 and transmitter electrodes 220 areseparated by insulative material disposed between them at cross-overareas; in such constructions, the electrode junctions (or pixels) may beformed with jumpers connecting different portions of the same electrode.In some embodiments, the transmitter and receiver electrodes areseparated by one or more layers of insulative material. In some otherembodiments, the transmitter and receiver electrodes are separated byone or more substrates; for example, they may be disposed on oppositesides of the same substrate, or on different substrates that arelaminated together. Moreover, one or more of the sensor electrodes canbe used for both capacitive sensing and for updating the display.Alternatively, the sensor electrodes may be implemented in a singlelayer design where the sensor electrodes do not overlap in the sensingregion. In some embodiment, the transmitter and receiver electrodes maybe separated by a compressible and/or deformable insulating material. Insome embodiments, the transmitter and receiver electrodes may beseparated by a uniform or individually spaced layer of force sensingresistors (FSR) or a piezoelectric material.

A capacitive image and a force image may be determined from measurementsreceived with the receiver electrodes. As noted above, the embodimentsof the invention can be implemented with a variety of different typesand arrangements of capacitive sensor electrodes or a variety ofdifferent non-capacitive proximity and touch sensing devices. Forexample, the electrodes for sensing may be disposed in a first direction(e.g., the “X” direction), a second direction (e.g., the “Y” direction),or in any suitable orthogonal, parallel, or hybrid configuration such aspolar coordinates (e.g., “r” and “0”). In these embodiments the sensorelectrodes themselves are commonly arranged in a circle or other loopedshape to provide “0”, with the shapes of individual sensor electrodesused to provide “r”. In other embodiments, the sensor electrodes may beformed on the same layer, or the input device can be implemented withelectrode arrays that are formed on multiple substrate layers.

Also, a variety of different sensor electrode shapes can be used,including electrodes shaped as thin lines, rectangles, diamonds, wedge,etc. Finally, a variety of conductive materials and fabricationtechniques can be used to form the sensor electrodes. As one example,the sensor electrodes are formed by the deposition and etching ofconductive ink on a substrate.

Again, it should be emphasized that the sensing electrodes 200 are justone example of the type of electrodes that may be used to implement theembodiments of the invention. For example, some embodiments may includemore or less numbers of electrodes. In other examples, the electrodesmay be formed on multiple layers. In yet other examples, the electrodesmay be implemented with an array of electrodes that have multiple rowsand columns of discrete electrodes.

FIG. 3 is a conceptual layout for a touch and force sensor stack-up 300including a touch sensor assembly 301 and a force sensor assembly 303,where the touch and force sensor assemblies share a common transmitterelectrode layer 306. Alternatively, the touch and force assemblies mayeach include dedicated transmitter electrodes. In the illustratedembodiment, the touch sensor assembly 301 includes a touch sensorelectrode layer 302 which cooperates with the transmitter electrodelayer 306 to produce a touch image. Similarly, the force sensor assembly303 includes a force sensor electrode layer 304 which cooperates withthe transmitter electrode layer 306 to produce a force image. Thoseskilled in the art will appreciate that various manufacturing andperformance efficiencies and advantages may be realized by sharing andstrategically arranging the various electrode layers and associatedsubstrate layers. For purposes of the present discussion, the sensorstack-up 300 may be configured in any desired manner to produce adiscretized (e.g., pixelated) touch image and a correspondingdiscretized force image. Thus, as described above, any non-capacitivesensing technique which produces a discretized force and/or touch imagemay utilize the techniques described herein to determine the positionand force applied by input objects

Turning now to FIGS. 4 and 5, exemplary input objects in a sensingregion are illustrated. In the illustrated embodiments, the inputobjects are fingers; alternatively, and as discussed below in connectionwith the force image segmentation process, the input object(s) maycomprise conductive objects (including hand parts such as fingers,thumb, and/or palm), non-conductive objects such as a fingernail,stylus, or pen, or a combination of conductive and non-conductiveobjects.

More particularly, FIGS. 4 and 5 show top and side views, respectively,of an exemplary input device having two fingers interacting with thesensing region. Specifically, FIG. 4 is a top view of an input device400 with respective input objects 404, 406 proximate an input surface402 of a sensing region. FIG. 5 is a side view of an input device 500showing one or more input objects 504, 506 applying force along adirection 510 to an input surface 502. As described below in greaterdetail, the touch sensor assembly produces a touch image of the inputobjects, and the force sensor assembly produces a corresponding forceimage (also referred to as a pressure image) of the input objects. Inaccordance with various embodiments, information obtained fromsegmenting the touch image may be used to optimize (or otherwisefacilitate) segmentation of the force image. Specifically, the touchimage may be segmented into mutually exclusive touch blobs, and thesetouch blobs may be used to seed initial force blob formation, therebyensuring reliable correlation between the resulting force blobs and therespective input objects which produced them.

Turning now to FIGS. 6 and 7, exemplary touch and force images areillustrated for multiple input objects. More particularly, FIG. 6depicts an array 600 of individual touch pixels 614 illustrating touchimage information for multiple input objects. Specifically, the touchimage includes a first group of pixels identified by the number “1”, asecond group of pixels identified by the number “2”, and a third groupof pixels identifies by the number “3”. Those skilled in the art willappreciate that robust algorithms have been developed for segmenting thepixels comprising a touch image into individual touch blobs, eachcomprising a unique subset of the touch image pixels. For presentpurposes, any one or combination of presently known or heretoforedeveloped techniques for touch segmentation may be employed inconjunction with the embodiments discussed herein.

With continued reference to FIG. 6, the touch image may be segmentedinto a first touch blob 601, a second touch blob 602, and a third touchblob 603, corresponding to respective first, second, and third inputobjects. More particularly, the first touch blob 601 includes thosetouch pixels—and only those touch pixels—denominated by the number “1”;touch blob 602 includes those pixels denominated by the number “2”; andtouch blob 603 includes those pixels denominated by the number “3”.Thus, the touch segmentation process assigns a unique subset of thetouch image pixels to a respective touch blob, with each touch blobcorresponding to a unique input object. Having determined which touchpixels correspond to each input object, it remains to determine theaggregate force applied by each input object. Importantly, though notshown in FIG. 6, the unique subsets of pixels corresponding to eachinput object may share at least part of one pixel or a whole pixel.

The force segmentation process assigns a unique subset of the forceimage pixels to respective force blobs, with each force blobrepresenting a unique input object. In accordance with variousembodiments, by using the determined touch blobs to seed the formationof force basins (or other computational construct) used to determineforce blobs, the relationship between individual input objects and theirassociated force information may be reliably determined.

More particularly, FIG. 7 is a top view of an array 700 of individualforce pixels 714 representing a force image of multiple input objects.Specifically, the force image includes a first group 701 of force pixelsand a second group 702 of force pixels. As discussed in greater detailbelow, each force pixel has an associated force value. Upon completionof the force image segmentation process, the force values of all pixelswithin each force blob may be summed or otherwise combined to determinethe applied force attributable to each input object.

With continued reference to FIG. 7, while each force pixel above acertain threshold value bears some relationship to at least one inputobject, the precise relationship between each input object and eachforce pixel may not be immediately apparent from the force image alone.Such ambiguity may be due to system noise, clumping of fingers together,and various motion and other artifacts. Accordingly, it is desirable tosegment the force image into one or more force blobs, with each forceblob corresponding to a unique input object in a manner analogous tosegmenting the touch image into touch blobs. However, independentlysegmenting the force image without reference to the segmented touchimage may result in ambiguities, particularly with regard to pixels nearclumped fingers which could arguably be allocated on one finger or theother. As explained more fully below, seeding the formation of forceblobs with touch blob pixels mitigates such ambiguity. In this regard,although each resulting segmented force blob may include more than, lessthan, or the same pixels as a previously determined touch blob, it isnonetheless desirable to use the touch blobs to seed the forcesegmentation process. In this way, the correlation between a particularinput object and its associated touch pixels, on the one hand, and theforce pixels associated with that same input object, on the other hand,may be preserved.

Referring now to FIGS. 6 and 8, touch image segmentation will now bebriefly described in greater detail as a predicate to the ensuingdiscussion regarding the use of touch blobs to seed force basins duringthe initial formation of force blobs.

FIG. 8 is a two-dimensional graph 800 of exemplary touch basins 802 and804 illustrating a touch segmentation process. In the present context, abasin (or “catchment basin”) is a graphical construct used to aidvisualization of an image segmentation process whereby the “depth” ofthe basin represents the magnitude of the change in capacitance measuredby the sensor electrodes. A typical segmentation process employs acomputer implemented algorithm to identify those pixels in a touch image(such as the touch image shown in FIG. 6) having a variable capacitanceabove a threshold value, and assign each such pixel to one or more(e.g., when a pixel borders two unique neighboring input objects) inputobjects.

With continued reference to FIG. 8, those skilled in the art willappreciate that a typical touch image segmentation algorithm identifieslocal minima 806 and 808, and assigns the lowest local minimum (e.g.,local minimum 808) to a first basin (e.g. basin 804). Thereafter, thethen currently lowest value pixels are recursively evaluated in a“rising water level” process. If the then current pixel under inspectionis adjacent to a previously assigned pixel, it is assigned to the sametouch basin as the previously assigned pixel; otherwise, the thencurrent pixel is assigned to a new basin. When a pixel such as pixel 810is encountered which is adjacent to two different previously assignedpixels, the algorithm uses predetermined metrics (such as the relativedepths of the two basins) to allocate the then current pixel, to eitherbasin or to both basins. Based on well-known metrics, the two basins maybe merged into a single basin; alternatively, a dam 812 may beconstructed to preserve the two basins during subsequent segmentationprocessing. In some embodiments, a dam may be formed “between” adjacentpixels, in other embodiments, a dam may be formed “through” a pixel,wherein the latter involves assigning the pixel to more than one basin.The basins are thus grown until all pixels above a threshold value havebeen assigned to at least one basin. Each fully segmented touch basin isthen declared a touch blob.

Once the touch image is segmented, the force image may then be segmentedinto force blobs. As discussed above, it is desirable to start the forcesegmentation process by seeding each force basin with a respective groupof touch blob pixels.

FIG. 9 illustrates an array of force sensor pixels showing the seedingof touch blob information into initial force basin formation. Forclarity, the touch pixels and force pixels are aligned on a one-to-onebasis, although the force pixilation may alternatively be greater thanor less than the touch pixilation.

More particularly, FIG. 9 depicts a pixelated sensor array 900 definedby X coordinates x₁-x₈ and Y coordinates y₁-y₈. In the illustratedexample, the touch segmentation process determined three distinct touchblobs: i) a first touch blob including a first subset of pixels {(x₂,y₂), (x₁, y₃), (x₂, y₃), (x₃, y₃), (x₁, y₄), (x₂, y₄), (x₃, y₄), (x₁,y₅), (x₂, y₅)}; ii) a second touch blob including a second subset ofpixels {(x₃, y₁), (x₄, y₁), (x₅, y₁), (x₃, y₂), (x₄, y₂), (x₅, y₂), (x₄,y₃)}; and iii)) a third touch blob including a third subset of pixels{(x₈, y₄), (x₇, y₅), (x₈, y₅), (x₇, y₆)}.

With continued reference to FIG. 9, regions 901 and 902 areadjacent/neighboring force pixels which have a value above a threshold.In various embodiments, force image segmentation begins with: i) seedinga first force basin 903 with the pixels from the first touch blob; ii)seeding a second force basin 904 with the pixels from the second touchblob; and iii) seeding a third force basin 905 with the pixels from thethird touch blob. As discussed in greater detail below, the third forcebasin is filtered out inasmuch as the pixels within the third forcebasin do not comprise any force pixels above a threshold.

FIG. 10 illustrates a cross-sectional view of a touch and force image,for the purposes of segmentation. A touch image cross section 1001includes a first touch blob 1002 and a second touch blob 1004 separatedby a dam 1012, and a force image cross section 1003 includes a firstforce basin 1002 and a second force basin 1004 separated by a dam 1030,as well as a third force basin 1032. Force image segmentation begins byseeding the first force basin 1022 with the pixels from the first touchblob 1002 and seeding the second force basin 1024 with the pixels fromthe second touch blob 1004. Thus, the first region 901 (FIG. 9) isinitially segmented into the first and second force basins 1022 and1024, subject to further force image segmentation processing asdiscussed below. Additionally, because there is no touch blobcorresponding to the second force image region 902, the force basinassociated with region 902 is not initially seeded; rather, it is grownorganically in accordance with the force segmentation algorithm as alsodescribed below.

With continued reference to FIG. 10, after initial seeding of the firstforce basin 1022 with the pixels from first touch blob 1002, the forcesegmentation algorithm proceeds to evaluate each force image pixel todetermine if it should remain within the assigned force basin. That is,if the then current force image pixel under inspection is adjacent to apreviously assigned force pixel, it is assigned to the same force basin;otherwise, a new force basin is begun. This is depicted graphically asforce basin 1032 which corresponds to the force image region 902. To theextent a force basin includes pixels in addition to those initiallyseeded from a touch blob, the force basin is grown accordingly as shownby the force basin region 1040 (corresponding to pixel (x₃, y₅)) andforce basin region 1042 (corresponding to pixel (x₅, y₃)). For thoseforce image pixels (such as pixel (x₄, y₄)) which are not assigned to atouch blob, the force segmentation may allocate the pixel to a forcebasin based on suitable image segmentation metrics, as in connectionwith the touch segmentation metrics for resolving ambiguity, for exampleassigning the pixel to both force basins, or assigning a part of thepixel to each force basin.

In the interest of completeness, a third force basin (not shown) mayalso be initially seeded with pixels {(x₈, y₄), (x₇, y₅), (x₈, y₅), (x₇,y₆)} from the third touch blob (FIG. 9), but during subsequent forcesegmentation processing this third force basin is reduced to the nullset because the force image does not contain any force pixels above athreshold which correspond to the touch blob for those pixels.

Referring now to FIG. 11, it can be seen that the force segmentationalgorithm may yield force blobs which include a greater or lesser numberof touch pixels than initially used to seed the corresponding forcebasin. More particularly, FIG. 11 a two-dimensional graphic illustrationof a touch blob 1101 and a corresponding force segmentation graphic 1103is shown. In the illustrated example, the force segmentation graphic1103 includes an exemplary first force blob 1104 of lesser extent thanthe touch blob 1102 (i.e., having a lesser number of total pixels), andan exemplary second force blob 1106 of greater extent than the touchblob 1102.

With continued reference to FIG. 11, a force basin is initially seededwith the pixels comprising the touch blob 1102, which may be graphicallyrepresented by a force basin region 1130 between a left border 1108 anda right border 1114; that is, the initially seeded force basin isdefined by that portion of the force basin region 1130 between borders1108 and 1114. In the event the force image segmentation process growsthe initially seeded force basin by adding additional force pixels intothe force basin (such as pixel (x₃, y₅) in FIG. 9), the resulting forcebasin may increase in size as shown by regions 1120 and 1122.Conversely, force image pixels (e.g., regions 1116 and 1118) may removedfrom the force basin during force image segmentation if, for example,the seeded pixels have a corresponding force value below a threshold,thereby yielding a smaller resulting force blob 1104.

Although the various embodiments described herein are not limited to anyparticular force image segmentation process or algorithm, the forcesegmentation metrics set forth in FIG. 12 may nonetheless provide usefulguidance in configuring a force image segmentation algorithm.

More particularly, FIG. 12 is a flow diagram 1200 of an exemplary forcesegmentation process, where the steps are not necessarily orderedsequentially. The force image segmentation process includes the stepsof: i) seeding at least one force basin with information determined fromthe touch image segmentation stage (Task 1202); ii) grow the seededforce basins with additional force image pixels according to a nominalimage segmentation algorithm (Task 1204); iii) merge adjacent forcebasins according to segmentation algorithm criteria only if the adjacentforce basins were not seeded from different touch blobs (Task 1206); iv)if the force pixel under examination is not adjacent to or neighboringan existing force basin, then start a new force basin (Task 1208); v)force pixels having no corresponding touch blob are deemednon-conductive and assigned their own force basin (Task 1210); and vi)touch blob pixels having no corresponding force image pixels above athreshold value (e.g., region 902 of FIG. 9) are filtered out (Task1212).

Upon completion of the force image segmentation process, i.e., once theforce basins are fully grown, the completed force basins are declaredforce blobs. The total force associated with each force blob may then becomputed using any known or hereafter developed technique for summing orotherwise combining the individual force values of the force pixelswithin a force blob.

A processing system is thus provided for use with an input device of thetype including a sensing region overlapping an input surface, and anarray of sensor electrodes configured to form a plurality of proximitypixels and a plurality of force pixels. The processing system iscommunicatively coupled to the array of sensor electrodes, and isconfigured to: determine a proximity image indicative of positionalinformation for input objects in the sensing region, based on a variablecapacitance associated with each of the plurality of proximity pixels;determine a force image indicative of local deflection of the inputsurface in response to force applied by the input objects, based on avariable capacitance associated with each of the plurality of forcepixels; determine a group of proximity pixels from the proximity imagecorresponding to each input object in the sensing region; determine agroup of force pixels from the force image corresponding to eachdetermined group of proximity pixels; determine the force for at leastone input object based on the determined groups of force pixels; anddetermine the position of at least one input object based on thedetermined groups of proximity pixels.

In an embodiment, the determined groups of force pixels may be based ona planar alignment of the determined groups of proximity pixels relativeto corresponding force pixels for each input object; that is, thealignment may be based on a mapping of a respective one of thedetermined groups of proximity pixels to a corresponding group of forcepixels for each input object, wherein the proximity-to-force pixelmapping one-to-one, greater than one-to-one, or less than one-to-one.

In an embodiment, the array of proximity pixels and the array of forcepixels may share a common array of two-dimensional coordinate locations,and further wherein determining the group of force pixels comprisesseeding the group of force pixels with the coordinate locations of oneof the determined groups of proximity pixels.

In an embodiment, determining the group of force pixels from the forceimage corresponding to each determined group of proximity pixelsinvolves, for each force pixel in the force image above a thresholdvalue, assigning that force pixel to at least one of the determinedgroups of proximity pixels, wherein the threshold value may be apredetermined value, a dynamically configurable value, or zero.

In an embodiment, determining the group of force pixels furthercomprises removing seeded force pixels having a value less than thethreshold value, and determining the force associated with each inputobject comprises summing the force values associated with the pixelswithin each respective group of force pixels.

In an embodiment, the processing system may be further configured todetermine the pressure applied to the input surface by each input objectbased on the groups of proximity pixels and the groups of force pixels.

In an embodiment, the processing system may be further configured todetermine the groups of force pixels using an iterative imagesegmentation algorithm which evaluates the force image on apixel-by-pixel basis, where the algorithm may be a watershed algorithm.

In an embodiment, the processing system may be further configured todetermine the position of a non-conductive input object touching theinput surface based on a group of force pixels grown from thesegmentation algorithm and for which there is no corresponding group ofproximity pixels.

In an embodiment, the processing system may be further configured tocontrol a user interface action based on a determined coordinateposition and force associated with at least one input object.

In an embodiment, the processing system may be further configured to:transmit a sensing signal onto a first subset of the plurality of sensorelectrodes; receive a first resulting signal from a second subset of theplurality of sensor electrodes; and receive a second resulting signalfrom a third subset of the plurality of sensor electrodes; wherein thefirst resulting signal comprises effects of input object presenceproximate the input surface, and the second resulting signal compriseseffects of input object pressure onto the input surface.

An input device for an electronic system is also provided including: asensing region overlapping an input surface; a plurality of sensorelectrodes configured to form a plurality of proximity pixels and aplurality of force pixels; and a processing system communicativelycoupled to the sensor electrodes. The processing system may beconfigured to: determine a proximity image and a force image indicativeof input objects in the sensing region; determine a first group ofproximity pixels from the proximity image corresponding to at least afirst input object; determine a first group of force pixels,corresponding to the first input object, by: seeding a first preliminarygroup of force pixels with information from the determined group ofproximity pixels; and augmenting the first preliminary group of forcepixels with at least one force pixel from the force image which areabove a threshold and adjacent to the seeded first preliminary group offorce pixels.

In an embodiment, the processing system may be further configured to:determine a second group of proximity pixels from the proximity imagecorresponding to at least a second input object; determine a secondgroup of force pixels, corresponding to the second input object, by:seeding a second preliminary group of force pixels with information fromthe determined group of proximity pixels; and augmenting the secondpreliminary group of force pixels with force pixels from the force imagewhich are above a threshold and adjacent to the seeded secondpreliminary group of force pixels.

In an embodiment, the processing system is further configured to removeforce pixels from the first force basin which have an associated forcevalue less than a threshold value.

In an embodiment, the processing system is further configured to:determine a first position of the first input object based on the firstgroup of proximity pixels; determine a first force associated with theaugmented first force basin; and control a user interface action basedon the first position and the first force.

In an embodiment, the processing system is further configured to:transmit a sensing signal onto a first electrode of the plurality ofsensor electrodes; receive a first resulting signal from a secondelectrode of the plurality of sensor electrodes; receive a secondresulting signal from a third electrode of the plurality of sensorelectrodes; determine a variable capacitance associated with at leastone proximity pixel based on the first resulting signal; and determine avariable capacitance associated with at least one force pixel based onthe second resulting signal.

In an embodiment, the processing system is further configured todetermine a variable capacitance associated with each of the pluralityof proximity and force pixels by: transmitting a sensing signal onto afirst subset of the plurality of sensor electrodes; receiving a firsttype of resulting signal from a second subset of the plurality of sensorelectrodes; and receiving a second type of resulting signal from a thirdsubset of the plurality of sensor electrodes.

In an embodiment, the processing system is further configured to:determine a position of the first input object based on the firstdetermined group of proximity pixels; determine a force of the firstinput object based on the first determined group of force pixels; andcontrol a user interface action based on the determined position and thedetermined force of the first input object.

In an embodiment, the processing system is further configured todetermine a force associated with the first input object based on forcevalues of the first determined group of force pixels, and determine aforce associated with the second input object based on force values ofthe second determined group of force pixels.

In an embodiment, the plurality of sensor electrodes comprises a firstsubset, a second subset, and a third subset of sensor electrodes, thefirst and second subsets correspond to the proximity pixels, and thesecond and third subsets correspond to the force pixels.

In an embodiment, the processing system is further configured todetermine a first force associated with the first input object bysumming force values within the first force basin, and determine asecond force associated with the second input object by summing forcevalues within the second force basin.

A method is also provided for determining the force applied to apixelated capacitive input surface by at least one an input object. Themethod involves: determining a proximity image and a force image for theat least one input object; determining a unique group of proximitypixels for each of the at least one input object from the proximityimage; determining a unique group of force pixels for each of the atleast one input object based on the unique group of proximity pixels;and determining a force value for the at least one input objects basedon the unique groups of force pixels; wherein each force pixel in theforce image above a threshold value is assigned to a single unique groupof force pixels.

The method may also include controlling a user interface action based onthe determined force value of the at least one input object.

The embodiments and examples set forth herein were presented in order tobest explain the present invention and its particular application and tothereby enable those skilled in the art to make and use the invention.However, those skilled in the art will recognize that the foregoingdescription and examples have been presented for the purposes ofillustration and example only. The description as set forth is notintended to be exhaustive or to limit the invention to the precise formdisclosed.

What is claimed is:
 1. A processing system for use with an input devicehaving an array of sensor electrodes, the processing system configuredto be communicatively coupled to the array of sensor electrodes andconfigured to: operate at least a subset of the array of sensorelectrodes to determine a proximity image for input objects in a sensingregion of the input device; operate at least a subset of the array ofsensor electrodes to determine a force image indicative of localdeflection of an input surface of the input device; determine a firstgroup of proximity pixels from the proximity image corresponding to afirst input object in the sensing region to obtain a determined group ofproximity pixels; determine a first group of force pixels from the forceimage corresponding to the determined first group of proximity pixels toobtain a determined group of force pixels; determine the force for thefirst input object based on the determined group of force pixels; andperform a user interface action based on the force determined for thefirst input object.
 2. The processing system of claim 1, wherein theproximity image is determined from a change in a plurality of variablecapacitances formed by the subset of the array of sensor electrodes,wherein each variable capacitance comprises a proximity pixel.
 3. Theprocessing system of claim 1, wherein the force image is determined froma change in a plurality of variable capacitances formed by the subset ofthe array of sensor electrodes, wherein each variable capacitancecomprises a force pixel.
 4. The processing system of claim 1, furtherconfigured to: determine the position of the first input object based onthe determined group of proximity pixels.
 5. The processing system ofclaim 1, wherein the determined first group of force pixels is based ona planar alignment of the proximity pixels relative to the force pixelsfor the first input object.
 6. The processing system of claim 1, whereinthe determined first group of force pixels are based on a mapping of oneof the determined group of proximity pixels to a corresponding group offorce pixels for the first input object, wherein the proximity-to-forcepixel mapping comprises one of: i) a one-to-one; ii) greater thanone-to-one; and iii) less than one-to-one.
 7. The processing system ofclaim 1, wherein the array of proximity pixels and the array of forcepixels formed by the array of sensor electrode share a common array oftwo-dimensional coordinate locations.
 8. The processing system of claim7, wherein determining the first group of force pixels comprises seedingthe first group of force pixels with the coordinate locations of theproximity pixels in the determined group of proximity pixels to obtain aplurality of seeded force pixels.
 9. The processing system of claim 1,wherein determining the first group of force pixels from the force imagecorresponding to the determined group of proximity pixels comprises: foreach force pixel in the force image above a threshold value, assigningthat force pixel to the determined group of proximity pixels.
 10. Theprocessing system of claim 8, wherein determining the first group offorce pixels further comprises removing seeded force pixels, in theplurality of seeded force pixels, having a value less than a thresholdvalue.
 11. The processing system of claim 1, wherein the determinedforce for the first input object is based on a force value associatedwith each force pixel of the determined group of force pixels.
 12. Theprocessing system of claim 11, further configured to determine pressureapplied to the input surface by the first input object based on acontact area of the first input object and the determined force for thefirst input object.
 13. The processing system of claim 9, whereinassigning each force pixel in the force image comprises using an imagesegmentation algorithm which evaluates the force image based on thedetermined group of proximity pixels for the first input object.
 14. Theprocessing system of claim 1, further configured to: determine a secondgroup of force pixels from the force image having no correspondingdetermined group of proximity pixels; and determine a position of anon-conductive input object contacting the input surface based on thedetermined second group of force pixels and the absence of thecorresponding group of proximity pixels.
 15. The processing system ofclaim 1, wherein performing the user interface action comprises updatinga display of the input device.
 16. The processing system of claim 1,wherein operating at least the subset of the array of sensor electrodesto determine a proximity image and a force image comprises: transmittinga sensing signal onto a first electrode of the plurality of sensorelectrodes; receiving a first resulting signal from a second electrodeof the plurality of sensor electrodes; and receiving a second resultingsignal from a third electrode of the plurality of sensor electrodes;determining the change in the variable capacitance associated with atleast one proximity pixel based on the first resulting signal; anddetermine the change in the variable capacitance associated with atleast one force pixel based on the second resulting signal.
 17. Theprocessing system of claim 16, wherein the change in the variablecapacitance associated with at least one proximity pixel occurs inresponse to the presence of input objects in the sensing region and thechange in the variable capacitance associated with at least one forcepixel occurs in response to the local deflection of the input surface.18. An electronic system having an input device, the input devicecomprising: a sensing region overlapping an input surface; a pluralityof sensor electrodes configured to form a plurality of proximity pixelsand a plurality of force pixels; and a processing system configured tobe communicatively coupled to the plurality of sensor electrodes andconfigured to: determine a proximity image and a force image indicativeof at least one input object in the sensing region; determine a firstgroup of proximity pixels from the proximity image corresponding to afirst input object of the at least one input object to obtain adetermined first group of proximity pixels; and determine a first groupof force pixels, corresponding to the first input object to obtain adetermined first group of force pixels.
 19. The electronic system ofclaim 18, wherein the processing system is configured to determine thefirst group of three pixels by: seeding a first preliminary group offorce pixels with information from the determined group of proximitypixels; and augmenting the first preliminary group of force pixels withat least one force pixel from the force image which are above athreshold and adjacent to the seeded first preliminary group of forcepixels.
 20. The electronic system of claim 18, wherein the processingsystem is further configured to: determine a second group of proximitypixels from the proximity image corresponding to a second input objectto obtain a determined second group of proximity pixels; determine asecond group of force pixels, corresponding to the second input object,by: seeding a second preliminary group of force pixels with informationfrom the determined group of proximity pixels; and augmenting the secondpreliminary group of force pixels with force pixels from the force imagewhich are above a threshold and adjacent to the seeded secondpreliminary group of force pixels.
 21. The electronic system of claim18, wherein the processing system is further configured to: control auser interface action based on the determined position and thedetermined force of the first input object by updating a display of theelectronic system.
 22. The input device of claim 21, wherein the inputsurface overlaps the display.